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I. Introduction 


The purpose of this study was to expose a range of naive individuals to the NASA 
Data Archive and to obtain feedback from them, with the goal of learning how useful 
people with varied backgrounds would find the Archive for research and other purposes. 

We processed 36 subjects in four experimental categories, designated in this report 
as C+R+, C+R-, C-R+ and C-R-, for computer experienced researchers, computer 
experienced non-researchers, non-computer experienced researchers, and non-computer 
experienced non-researchers, respectively. 

This report includes an assessment of general patterns of subject responses to the 
various aspects of the NASA Data Archive. Some of the aspects examined were interface- 
oriented, addressing such issues as whether the subject was able to locate information, 
figure out how to perform desired information retrieval tasks, etc. Other aspects were 
content-related. In doing these assessments, answers given to different questions were 
sometimes combined. This practice reflects the tendency of the subjects to provide answers 
expressing their experiences across question boundaries. Pa items of response are cross- 
examined by subject category in order to bring out deeper understandings of why subjects 
reacted the way they did to the archive. 

After the general assessment, there will be a more extensive summary of the replies 
received from the test subjects. The raw data sheets are provided as Appendix A.. 

II. Classification Categories 

The subject pool was divided into four groups along two main divisions: computer 
experienced and research experienced. The intent was to use the computer experience axis 
to distinguish interface-related issues as seen from a novice perspective to those as seen 
with a more experienced eye. Often, problems that are daunting to the novice are a matter 
of course for those more comfortable or practiced with computers and data retrieval. The 
research experience axis was intended to isolate those subjects with the background to 
provide informed assessments of the quality of the contents of the archive and the 
suitability of those contents for scientific research purposes. Non-researchers' content 
feedback was solicited partially for symmetry and partially because it provides insight into 
how non-scientists might react to and use the Data Archive. 

For purposes of this study, a computer-experienced (C+) individual was one who 
had used a range of different computer programs determined to indicate a greater than 
casual familiarity with computers. As many people use word processors to write papers, 
Quicken or similar software to track their finances, etc., and perform these tasks more by 
rote than by understanding, the significance of such programs as signs of computer 
experience in the relevant sense was discounted. C+ subjects typically were experienced in 
a range of different computer programs, including databases, statistical analysis software, 
Internet software, spreadsheets, graphics programs and more. 

Researchers in this study were those individuals who had familiarity and experience 
with the process of scientific research, including the gathering of data, the formulation of 
hypotheses and the testing of them against evidence, the use of statistics and other standard 
tools of modem scientific research. Researchers were those capable of providing infonmed 
feedback on the utility of the information in the Data Achive for research purposes. 

III. Experimental Design 
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The Data Archive evaluation was conducted using 36 subjects, divided into four 
categories of nine people each, as described above. Each subject was provided with a set 
of response forms and a small set of compulsory tasks. The subject then spent 90 minutes 
working with the Data Archive, initially trying to accomplish the compulsory tasks, then 
just exploring as his or her interest dictated. At the end of the 90 minutes, each subject 
filled out the response forms, describing their experience with the Data Archive and their 
evaluation of it Subjects were paid $20.00 each for their participation. 

Individuals were recruited into the subject pool through a variety of methods, 
including the posting of flyers around the San Jose State University campus, placing ads in 
the campus newspaper, and through personal contacts. 

IV. Problems 

There were a number of confounding factors which complicate interpretation of our 
results. The most significant of these problems was the omission, by oversight, of the raw 
data sets from the archive when the archive was loaded onto our computer s local hard 
drive. This omission prevented subjects from retrieving any Excel spreadsheets or raw 
data while exploring the data archive. As the raw data omitted was arguably one of the 
primary strengths of the archive as it related to research, this error significantly impacted 
the assessment of the utility of the archive for research. Oddly, a number of subjects 
provided responses that, on face value, implied that they were able to retrieve spreadsheets 
and/or raw data. The most likely explanation is, in the face of questions that obviously 
presupposed the existence of a named entity, subjects assumed that that name applied to 
something they were able to find. This hypothesis is partially supported by the fact that the 
more inexperienced subjects (in both computer and research backgrounds) tended to make 
these replies disproportionately. 

We plan to address this weakness in the study by asking back a number of our test 
subjects with strong research backgrounds to assess a version of the archive that does 
contain the previously omitted data. Their replies will be contrasted with the "no raw data" 
responses to see how much of the negative reaction can be obviated by presenting the data 
archive in its more complete form. When available, these data will be added to this report 
as Appendix B. 

Some less significant problems may have stemmed from the structure of the study 
itself. The first problem stemmed from the section of the study called "Compulsory 
Tasks”. The intent of the compulsory tasks was to ease the subjects into using the archive 
to perform some simple tasks, to provide a baseline and to ease the learning curve. 
However, the contents of the Data Archive changed between the time the compulsory tasks 
were generated and the testing began. As a result, many subjects got stuck trying to 
perform compulsory tasks that had either become much more difficult than intended or 
literally impossible. (For instance, the omission of the raw data sets rendered one 
compulsory question impossible to complete properly.) This had a negative impact on the 
assessment of the interface by a number of subjects. 

The second problem resulted from poorly phrased questions, particularly in the 
content assessment portion of the questionnaire. Subjects were asked "would this archive 
be useful to you in your work", when the question that should have been asked was more 
along the lines of "would this archive be useful to someone trying to perform scientific 
research". A number of subjects rated the utility of the archive "for their work" very low, 
but went on to comment "This would be great for someone in a different research field." 


3 



It was often possible to identify subject responses that had been tainted by these 
flaws in design and implementation, and where possible these were called out in the report. 
These flaws should be kept in mind while assessing the results of the study as a whole. 
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V. Results Summaries 

A) Computer Experienced Researchers (C+R+) 

Results from the C+R+ group are applicable to two of the three types of issues 
being examined by the study -- the reaction of computer knowledgeable people to the 
archive interface, and the assessment by researchers of the archive’s content 

In general, the interface rating given by C+R+ individuals was fairly stable, ranging 
from 2 to 5 (7 high) with one abstention. The most prevalent complaint had to do with the 
lack of clarity of button labels. Subjects were often not clear what kinds of information 
would be found in what sections of the database. This complaint was also expressed by 
other groups, but it is particularly telling coming from this group, given their regular use of 
computers as research tools. The second most prevalent complaint (also strongly reflected 
across group boundaries) was a cluster of problems relating to search. Subjects 
complained that it was unclear which search fields had popup menus to select from, that the 
popup menus were difficult to use, and that the results were often counterintuitive. (Two 
examples of the latter: One subject complained that a search turned up "no match" on a term 
that he later saw while browsing other items. Another complained that the search 
categories themselves were sometimes difficult to interpret.) One suggestion to improve 
search that came up in multiple groups was a capacity for full-text search indexing, 
allowing free text entry that would be matched against full data text in a field-independent 
manner. 

Other minor problems reported were irritation that various buttons (most notably the 
"Go Back" button) would scroll off of the screen as the subject read down, e.g. an 
experiment description. One subject noted that you don't want to have to scroll back up to 
the top of the screen if you're just going to go back from there. A number of subjects also 
expressed confusion with the profusion of space-related acronyms in the archive, and 
found it irksome to have to move to a separate portion of the database to get expansions. A 
popup acronym definition would go far to alleviate this problem. 

Virtually all C+R+ subjects were (or would have been) able to retrieve data sets for 
experiments. (2 answered "Yes", 2 said "they figured it out but couldn't do it because the 
sets weren't there" and 4 said they could not "because the sets were locked out", which 
implies that they did figure it out but had got bitten by the absence of the data sets.) 

No subject provided feedback about difficulties in the data presentation. This can 
be interpreted either as meaning that they all found the presentation clear, or that they all 
skipped it since they were not able to retrieve raw data. 

Collectively, the subjects used virtually all of the functions of the archive, however 
they were unable to access the Glossary due to a password protection that should be 
removed. 

Functions that ranked highly as unclear included "Subjects" (possibly interpreted as 
meaning "What experiments were about” instead of "What experiments were done on"), the 
Acronym definitions, and the unusable Glossary. 

The content suitability feedback is somewhat ambiguous. In raw terms, the utility 
of information was rated covering the full range of possibilities from 1 through 7. 
However, when comparing this with subject elaborations, we find that two of the l’s, a 2 
and a 3 were coupled with comments to the effect that "the information doesn't match my 
field", plus one 3 coupled w'ith the comment "would be better with raw data”. This seems 
to indicate that the level of content is belter than the numbers indicate when adjusted for 
matches between field and content. (Since some of the field mismatches were to 
researchers in fields like geology and ecology, the mismatching may not reflect the range of 
data in the archive so much as the type of experiments it is presently reasonable to perform 
in space.) A similar pattern can be seen in the "content level" assessments. The lower 
rankings came from the same subjects who rated the "work suitability" low due to content- 
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field mismatches. The abstention came from an individual withholding judgment in the 
absence of the raw data sets. 

Overall, the C+R+ subjects seemed to find the amount of information in the archive 
to be appropriate, neither overwhelming nor lacking, with typical responses falling in the 4- 
5 range on the 7 high scale. Hardware descriptions were seen as weaker than other 
components of the archive. The data set/element description were viewed as much too little 
information, probably due to the lack of the raw data, although 3 subjects rated them in the 
4-5 range. These higher ratings seem to reflect an assessment of the data apart from the 
missing data sets. 

The assessments of the Excel spreadsheets varied widely, which is odd, since the 
spreadsheets weren't present in the archive. The assessments are bipolar in nature, with 
2/3 of the subjects providing no feedback, and the other 1/3 rating them very highly. The 
assumption is that die 1/3 who rated the non-existent spreadsheets were in fact rating some 
other aspect of the archive that they had mistaken for the spreadsheets, but what that aspect 
might be is unknown. 

In the final analysis, the C+R+ group rated the archive as generally not useful for 
scientific work, with responses falling mostly in the 1-3 range. However, this result 
should be taken cautiously since it was severely impacted by the absence of the raw data 
sets. The prevalence of comments asking for more data, the missing data sets, and other 
such variants provide grounds for thinking that were the extra data available, the ratings 
could change significantly. This claim will be verified by the follow-up study mentioned in 
the Problems section above. 

B) Computer Inexperienced Researchers (C-R+) 

Results from the C-R+ group are also applicable to two of the three issues being 
examined by this study - the reaction of computer inexperienced people to the archive 
interface, and the assessment by researchers of the archive's content. 

In general, as with the C+R+ subjects, the interface rating given by the C-R+ group 
was fairly stable, centering around 4 with one 2 and one 7 on the 7 high scale. Overall this 
group found the interface somewhat more difficult than the C+R+ group, which isn't 
unreasonable considering they were less experienced computer users. They did not by- 
and-large find it impossible to use, however. 

The most prevalent interface problems with this group centered around the search 
screen and the lack of clarity of the catalog labels. One subject, for example, found the 
distinction between "Subjects” and "Biospecimens" to be unclear. Many users got hung up 
trying to enter search criteria on the search screens, complaining about the difficulty of 
using the popup menus and the inconsistent interface. 

Other interface complaints paralleled the C+R+ group in general outline - the lack 
of familiarity with terminology, especially acronyms; the popup menus; "no match" results 
from searches selected from menus, etc. Some subjects found the help section to be 
insufficiently helpful, saying that it stopped too soon, giving inadequate depth. 

More subjects indicated that they were able to retrieve data than the C+R+ group, 
which again should be interpreted either as subjects mistaking various types of summary 
for raw data, or as subjects figuring out how to retrieve data but being unable to do so. 

The former is a belter assessment than the latter, given the prevalence of comments like 
"had trouble figuring out where to look". 

C-R+ subjects used the full range of buttons and functions while exploring the 
catalog. Problems encountered included the lack of depth of the help functions, the lack of 
clarity of catalog labels and the tendency of buttons (also mentioned by other groups) to 
scroll off-screen. Placing key buttons in a static area that remains accessible might be a 
good idea even if it reduces the screen available for scrolling data. 
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The content suitability rankings were much lower overall than they were for the 
C+R+ group. This is most likely a result of the type of researchers used in the pool. 
Computer inexperienced life science researchers are difficult to locate, and thus the 
composition of the C-R+ pool is skewed in favor of other softer research areas, such as 
psychology, anthropology and environmental science. These research disciplines have 
very little overlap with the content of the archive, and as such the work suitability and 
background appropriateness tended to be poor. A number of replies indicated that the 
information was not useful to them due to field differences. 

Overall, the data content of the archive was viewed as acceptable to skimpy, with 
one dissenter who found it overly full/intimidating. Again, the hardware descriptions were 
viewed as particularly inadequate, as were the document descriptions. In general, the 
content was rated slightly more poorly than it was by the C+R+ group, which could 
indicate a greater problem in accessing data. One subject commented that his "other 
desired” additions to the archive would be "more step-by-step instructions". As with the 
C+R+ group, the missing data sets were missed, and it is very possible that a re- 
examination of the archive in its complete form would result in different assessments of its 
content suitability. 

The response to the non-existent spreadsheets was similar to that of the C+R+ 
group, with the bulk of the subjects declining to answer, but a significant minority rating 
something. 

The subjects in general found the archive suitable for preliminary scientific 
research, possibly as a pointer to more comprehensive data in the literature. This reaction 
makes sense given the lack of raw data sets, coupled with the poor overlap between the 
content and the background of many of the subjects. 

One subject wanted full listings of documents and resulting publications and more detailed 
equipment information. Requests for contact information for investigators were also 
received. 

Subjects suggested as helpful changes improving the placement of the "Go Back" 
button at the top of the scrolling pages and the addition of a free text based search system. 
These suggestions parallel ones made by other groups. 

C) Computer Experienced Non- Researchers (C+R-) 

Results from the C+R- group are applicable to only one of the three types of issues 
being examined by this study - the reaction of experienced computer users to the archive 
interface. Lacking significant research experience, the detailed assessments of these 
subjects regarding the content suitability of the archive for research purposes is being 
discounted, although salient comments will still be examined with an eye to seeing what the 
intelligent layman makes of the archive content. 

In general, the ease of use of the interface by the C+R- group exhibited a rough 
bipolar distribution, with 2/3 rating the interface between 1 and 3, the other 1/3 between 5 
and 6 on the 7 high scale. Of the two individuals rating the interface as very difficult, one 
seemed to be stuck as much on content as on interface (as judged by such comments as 
"For someone who could understand and be interested in it, it would be great."). The 
other's feedback is ambiguous. He rated the interface as very difficult to use, then 
peppered his comments with expressions like "very easy to use." Possibly the subject 
misread the ranking scale and thought 7 was "easiest to use" rather than "most difficult"? 

As with the other groups, the most commented on problems centered around the 
search facilities, with subjects asking for free text search capability and complaining about 
the unclear labeling of catalogs, components, and search fields. 

The lack of experimental raw data sets was criticized, not surprisingly, as was the 
seeming necessity of remembering cryptic experimental ID codes to plug into other search 
screens. Some subjects make suggestion that point to the direction of a more hypertext- 
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style interface, e.g. wishing to click on an experiment ID and move directly to the write-up 
of that experiment. 

Not surprisingly, the C+R- subjects had a poorer grasp of the nature of 
experimental data, which resulted in them misidentifying summaries as raw data. Subjects 
were able to locate data, however, to the extent that it was present in the archive. Subjects 
were sometimes confused by the seeming lack of organization of the data, by the lack of 
data sets, and by the prevalence of jargon and code numbers in the archive. 

C+R- subjects collectively used the full range of buttons and functions. Complaints 
centered around the non-functioning Glossary and the tendency of the "Go Back” button to 
scroll off screen. 

Being non-researchers, the C+R- group rated the archive as fairly unsuitable for 
their work, with two types of exception. The first type consists of students in biological 
sciences, who are too young to have research experience as yet These students 
conceptualized their "work” as their schoolwork and stated that they would find the 
database useful in that context. The second type consists of non-biological sciences 
individuals who simply found the contents interesting. 

In general, the assessments of content centered around "acceptable". 

Subjects requested contact methods for investigators, such as postal mail addresses, 
e-mail addresses, and telephone numbers. Some non-technical students found the amount 
of information to be intimidating and thought a more "layman oriented" style of information 
would be useful if the archive were to be targeted outside the research community. 
Requested changes includes free text searching, a better help system, and page-to-page 
navigation improvements. 

Subjects were of the opinion that the content would be useful to scientists, but this 
should be taken with a grain of salt given their lack of research experience. Potential 
methods for using the archive included as a general overview of what NASA has been 
doing, as a sign of what kind of research is being done in a given field. 

D) Computer Inexperienced Non-Researchers (C-R-) 

Results from the C-R- group are applicable to one of the three types of issues being 
examined by this study - the reaction of computer inexperienced people to the archive 
interface. Since these subjects lack the research experience necessary to make informed 
judgments about the applicability of archive content to research purposes, their comments 
on this issue are being discounted. General reactions to content will still be included, as 
with the C+R- group. 

In general, the interface was viewed as easy to understand and use. The single . 
exception had major problems with the search screens and found the help section to be "a 
waste of computer space." Subjects complained about the lack of clarity of the search 

screens, the acronyms and the non-working Glossary. 

Primary hang-ups occurred while trying to use search to retrieve information, and 
trying to retrieve experimental data. This latter problem is most likely a result of the 
missing data sets, compounded by the inexperience of the subjects with computers. There 
is a tendency among novices, when something doesn’t work, to keep trying it. The 
manner in which the archive failed while trying to recover the data sets was not very clear. 
Subjects assumed that the problem was with what they had done, and not with the archive, 
and this thus became a time sink for inexperienced subjects. The general lesson to draw 
from this is that failure modes should be clear. 

C-R- subjects complained about the inconsistency of the popup menus, joining the 
chorus from the other groups. They also indicated trouble locating data within the archive. 

Subjects used the full range of buttons and functions, finding unclear the hardware 
section, the data sets (unsurprisingly), the glossary and the acronyms. In general, the 
subjects in this group had trouble with the same areas as the other groups, but tended to get 
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stuck on them due to their lack of reference points. People with computer experience could 
use that knowledge as an anchor while exploring the archive; people with research 
experience could use their knowledge of the domain covered by the archive to orient 
themselves. Subjects in the C-R- group lacked either of these possible reference points, 
and thus sometimes found themselves lost in a sea of data not knowing what they were 
looking at or how to navigate. 

As non-researchers, the subjects rated the utility of the archive for their work very 
poorly, but this doesn’t reflect back on the general quality of the data for the reasons cited 
above. Virtually all the subjects qualified their poor rankings with remarks stating that their 
field is different from that covered by the archive, or that they aren’t scientists, etc. 

Overall, subjects tended to find the archive full of data, consistent with the 
overwhelming feeling of being faced with a mass of information in an unfamiliar domain. 
The document and data set descriptions were an exception to this, presumably as a result of 
the missing data sets. 

C-R- subjects were of the opinion that the content would be useful to scientists for 
research purposes, but qualified this assessment by noting that they themselves are not 
researchers. Subjects indicated the primary use of the archive for them would be as a 
general knowledge source, rather than as a directed tool. 

VI. Summary 

Some general patterns across groups were clearly noted._ Subjects in all groups, 
regardless of computer experience, found the search screens ditficult to use. The 
inconsistency of the popup menus was a cause of considerable frustration. The scrolling 
selection boxes didn't work well when they contained large numbers of selections, and the 
necessity of double clicking to select an element, then single clicking to make the menu go 
away, was very confusing and hard to work with. The frequent "no match" search results 
also caused problems, especially when subjects saw the rejected search term while 
browsing other documents. This violated the typical subject's intuitions about how search 
should work, causing many of them to advocate a free text search alternative that wouldn't 
be limited by hardwired classification fields. 

The plethora of acronyms and space jargon was also confusing to a number of 
subjects, both novices and experienced researchers. Even though many of these terms are 
defined inside the archive, having to break out of a search (often obtained after a fair 
amount of work) to get a definition for a term is very disruptive. Having to do it more than 
once is even worse. The suggestion of a popup term definition window was made by 
multiple subjects to address this problem. 

A number of subjects commented that the learning curve on the archive is about 10 
minutes. For that initial' 10 minutes, many felt lost, but after that, things settled down and 
they were able to work fruitfully. This implies that the greatest risk of losing a new person 
trying out the archive for the first lime is in that initial 10 minutes. Perhaps an expanded 
on-line help facility would address this potential problem. 

Most subjects were impressed with the range of data in the archive; the weaknesses 
cited in the data sets and document descriptions would be addressed by returning the 
deleted raw data to the archive. If that were done, the researcher groups seemed to indicate 
that the archive could become a useful research tool. 

A number of the less-experienced subjects indicated that the aichive should be 
"dumbed down" to make it more accessible to the average person. This could indicate a 
tension between having an archive that is sufficiently complete and technically rigorous to 
be useful for research, and an archive that will not overwhelm the student or layman with 
reams of complex data and analysis that get in the way of the basic ovci view sought by 



such individuals. The proper response to such feedback depends on the intended focus of 
the final archive, and as such is a policy decision beyond the scope of this study. 

VII. Summary of Questionnaire Responses by Group 

A) Computer Experienced Researchers (C+R+); N = 9 

From the Overall Evaluation Form: 

1. How easy was it to understand the interface, on a scale of 1 to 7 (7 hardest)? 

3, n/a, 2, 5, 5, 2, 5, 2, 4 

2. Which, if any, parts of those screens were unclear? 

1 Biospecimen and Hardware screens not well named 

1 Screens seemed clear but did not function as expected 

2 No answer 

2 Screen labels too short; had to use to figure out 
1 Search and sort screens 

1 Not sure if lists of experiment ID's were provided for retrieving information 
related to a certain experiment 

1 Unclear which search screen fields had popup menus 

3. While using the archive, where did you get stuck, lost or hung up? 

1 Personnel screen hangs up on first name entry 

2 Search screens 

4 No answer 

1 Help wasn't very helpful 

1 No glossary 

4. Did you have any trouble using the search screen and search function? 

1 Never knew what View List All Items would produce, e.g. 

Research Subjects list by ID, not by subject. 

1 Annoying that they look obvious but don't work well 

2 No answer 

1 Search criteria difficult to interpret; should have category- 
independent search field 
1 Trouble with most everything 

1 Sometimes searches would turn up "no match" on terms seen later 
while browsing other items 

1 Sometimes experiment ID's needed but not knows; more search choices 
needed 

1 Pop up menus troublesome, partial field matches and Boolean searches 
on single fields would be useful 


5. Please explain any other problems or suggestions you had while using the Data 
Archive. 
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1 Confusing acronyms; "Subject" implies something simpler than a complex 
code 

1 Sort has no clear method of subsorting; needs another level of search 
1 Straightforward once you get the hang of it 

1 "Go Back" button should stay on screen while scrolling; missed useful 
buttons at bottom of scrolling screens for some time 

1 Suggests using web-based access system, e.g. Netscape Directory Search 
model 

2 No answer 

1 Trouble figuring out terms and acronyms related to space 

1 Large scrolling text windows should have larger more readable fonts and 
margins. 

6. Were you able to find data from experiments? 

4 No, sets locked out (not present) 

2 Figured out how but wasn't able to actually retrieve data 

1 Yes, but unorganized 

1 Yes 

1 No answer 

7. Were you able to find data easily? 

5 No 

1 Yes, once past learning curve 

1 Sometimes 

1 Yes, up to the actual retrieval attempt 

1 No answer 

8. What about the data presentation was difficult to understand? 

9 No answer 

9. While browsing the catalogs what buttons and/or functions did you use? 

2 All 

1 Search, mission, experiment, data sets, subjects, hardware, personnel 

1 Search, help 

2 No answer 

1 Search. Tied glossary, sort, find all, buttons at end of some files 

1 Search, retrieve, go back, scroll bar, clicking for details 

1 All, although Glossary wanted a password 

10. Of those buttons and/or functions, which were not clear? 

1 Subjects; also locked out of Glossary 

1 Many clear but didn't function as name indicated 

1 Possible to figure out, but short synopsis/documentation of how to 
manipulate would be very useful. 

3 No answer 

1 AB = Aero (sp?); home vs. Go Back; Glossary' was unavailable 
1 None 

1 Glossary and AB= were unclear, as was Find All 



From the Content Suitability Form: 

a. How useful is this type of information to your work on a scale of 1 to 7 (7 most 

useful)? 

6, 2, 7, 1.2, 3. 1,7.3 

b. Is the content level right for your background (1 - 7, 7 most useful)? 

6, 2, 7, 6, 3, 5, 2, 3. n/a 

c. Please elaborate: 

Answers not statistically relevant here 

1 Descriptions were excellent, just like journal abstracts 

2 Other research fields would find info more useful 

1 Field not related 

1 To properly <- v Juate the use of an experiment, one needs a 
specific purpose 
1 Needs data sets 

d. Is there other information you would like to see provided... 

(Scale of 1 - 7, 1 is too little, 7 is too much) 

Overall 

4, 1,4, 4, 5, 5, 5, n/a, 4 

Experiment descriptions 

4, 1, 4, 3, 5, 5, 2, 2, 3 

Hardware descriptions 

4, 1, 1, 4, 5, 4, n/a, 2, 4 

Document descriptions 

3, 1,2, n/a, 5, 4, 4, n/a, 2 

Data set/element descriptions 

n/a, 1,1, n/a, 5, 4, 4, 1, 1 

Other desired 

4 No answer 

1 Experimental data; hard to get hardware descriptions 
1 Generally good amounts of information 
1 Experimental data; how data was treated for statistical 
conclusions 

1 Experimental data, purpose of experiments 

1 Data sets 

e. Any additional comments 

5 No answer 

1 Experimental descriptions excellent, hard to navigate, esp. hardware 
& data 

1 Info is good; access is clunky 
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1 

1 


Subject had fun 

Don't let "Go Back" button scroll off screen 


B . a. How well organized is the information on the Excel spreadsheets? 

(Scale of 1 - 7, 1 is not organized, 7 is well organized) 

n/a, n/a, 6, n/a, 6, 5, n/a, n/a, n/a 

b. How understandable is the information and data on the spreadsheets? 

(Scale of 1 - 7, 1 is not clear, 7 is very clear) 

n/a, n/a, 6, n/a, 6, 5, n/a, n/a, n/a 

c. Any additional comments on spreadsheet data? 

7 No answer 

1 Data superb, navigation tricky 

1 Data hard to obtain at first 

C. a. Are total data useful for scientific work? 

(Scale of 1 - 7, 1 is not useful, 7 is highly useful) 

3, n/a, 7, 1, 3, 3,.5, n/a, 3, n/a 
Comments 

3 Seems limited, perhaps due to lack of data sets 

4 No answer 

1 Information organized, but need to be able to move between related 
types so as to correlate and compare 

1 Less useful since experiments done in space, while subject is stuck 
on Earth 

b. How might data be useful to tester? 

4 No answer 

1 Researching areas of interest; finding out about hardware; learning 

who is 

doing research in a given area as a prelude to contact 
1 Reference for future experiments 

1 Doing reports on space research; ideas for new types of 
experiments; learning 
about NASA 

1 Not much overlap for subject’s particular interests 

c. Any other helpful changes? 

1 Sometimes items on popup menus don't come up in searches 
3 No answer 

1 A simple description of how to use system in one place would be 
useful; 

search was very useful one figured out 
1 Add balloon help 
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Use Netscape as model for searching; use threads, multiple search 
items. 

Boolean search terms 
1 A working glossary; field independent searching 
1 Better fonts in text windows; improve popup search menus 


Computer Inexperienced Researchers 
From the Overall Evaluation Form: 

1. How easy was it to understand the interface, on a scale of 1 to 7 (7 hardest)? 

4, 4, 2, 4, 5, 4, 7, 3, 2 

2. Which, if any, parts of those screens were unclear? 

4 The looking up categories screen 

3 Not obvious where to go to find particular pieces of info 

1 Search screens not accurate when doing query tasks 

1 Most of them 

3. While using the archive, where did you get stuck, lost or hung up? 

4 When user had to enter search data 

1 Search screens -- inconsistent clickable menus caused problems 

1 Finding experiments on specific research topics 

1 "Go to related data set" on experiment info page; unable to access glossary 

1 Hardware screen 

1 Searching for studies in the cardiovascular area 

4. Did you have any trouble using the search screen and search lunction ? 

1 Yes, due to unfamiliarity with life science terminology 

1 Yes, deciphering acronyms, restricting experiment searches to given 
mission 

2 Yes, tended to get "no match" due to need for exact match ot search terms 

1 Yes, popup menus difficult to use — double click required to select, then 

menu 

stays up; when "continue" is hit after a no match, still gives list of 
experiments 

2 Yes, trouble entering search terms 

2 Yes, "it didn't work". 

5. Please explain any other problems or suggestions you had while using the Data 
Archive. 

1 When asking for help it only went so far and then stopped providing info 
1 Easier access to acronym catalog 

1 Subject had trouble figuring out range of data contained in archive 

2 Buttons tended to scroll off screen 

1 "Scroll this way --->" marker confusing 

1 Would have preferred to explore first (problem with experiment setup) 

1 All mission screens had "movie" buttons, but not all had movies 
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1 No answer 

6. Were you able to find data from experiments? 

4 Yes 

3 No, trouble figuring out where to look 

2 No, data sets not present 

7. Were you able to find data easily? 

1 No; difficult in figuring out program w/o manual 

3 No, trouble figuring out where to look 

2 No, data sets not present 

1 No, didn't know where to start 

1 No, search wasn’t working 

1 Yes 

8. What about the data presentation was difficult to understand? 

1 Lack of written instructions; deciphering acronyms 

2 Explanations of what kind of stuff is under each catalog 

1 What search parameters will be accepted 

1 Data as ID#'s was confusing 

1 Movie data missing; help hard to follow; nature of experiments unclear 

3 No answer 

9. While browsing the catalogs what buttons and/or functions did you use? 

1 Mission, data sets, subjects, experiments, documents, hardware 
1 Mission, experiments, data sets, subjects, personnel 
1 All except Glossary and Sessions 

1 Search, sort, find all 

1 Back and home 

1 Mission, experiment, personnel, data, go back, scrolling, home 

1 Search, browse and go back 

1 Exit, retrieve document, perform search 

1 Movie was great 

10. Of those buttons and/or functions, which were not clear? 

1 The help functions 

1 Subject -- not clear what various subjects meant 

3 No' 

1 Buttons tend to scroll away 

1 Search very unclear 

2 No answer 


From the Content Suitability Form: 

A. a. How useful is this type of information to your work on a scale of 1 to 7 (7 most 
useful)? 


2, 1,2, 6, 2, 2, 1, 1,4 
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b. Is the content level right for your background (1 - 7, 7 most useful)? 

2, 1, 2, 7, 4, 2, 1, 1, 5 

c. Please elaborate: 

Answers not statistically relevant here 
1 Information detailed, easy to find 
3 Not useful due to field difference 
1 This level of data is only useful as a framework 

1 Info is scattered, but overall content is thorough 

d. Is there other information you would like to see provided... 

(Scale of 1 - 7, 1 is too little, 7 is too much) 

Overall 

2, 4, 4, 5, n/a, 2, 1, 7, 3 

Experiment descriptions 

2, 3, 3, 5, 4, 2, 3, 7, 3 

Hardware descriptions 

1,4, 4, 2, 1, 1,2, 7,4 

Document descriptions 

1, n/a, 4, 4, 1, 2, 2, 7, 2 

Data set/element descriptions 

2, 4, 4, 3, 1, 2, 3, 7, 2 

Other desired 

1 More step-by-step instructions 

3 Actual data sets 

1 Site where experiment look place 

1 More detail on hardware, data sets 

1 More experimental info (data sets?) 

2 No answer 

e. Any additional comments 

2 Found acronyms/terminology confusing 

1 Experiment description should be divided into sections, e.g. 

methods, results, 
discussion, etc. 

1 Missing data sets 

1 Missing data sets really hurl; keyword search would be nice 
1 Search software needs revamping 

3 No answer 

B. a. How well organized is the information on the Excel spreadsheets? 

(Scale of 1 - 7, 1 is not organized, 7 is well organized) 

6, 5, n/a, n/a, n/a, n/a, 4, 5, 3 
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b. How understandable is the information and data on the spreadsheets? 

(Scale of 1 - 7, 1 is not clear, 7 is very clear) 

2, 3, n/a, n/a, n/a, n/a, 2, 2, 2 

c. Any additional comments on spreadsheet data? 

1 Confused by scientific terms 

1 Needs to be clearer 

7 No answer 

C. a. Are total data useful for scientific work? 

(Scale of 1 - 7, 1 is not useful, 7 is highly useful) 

3, n/a, n/a, 5, 4, 3, 2, 3, 3 
Comments 

1 Once past interface, should be useful 
3 Seems sufficient for preliminary work/research 

1 Unsure of what type of work could be done 
1 Layout good, content good, access methods poor 

3 No answer 

b. How might data be useful to tester? 

1 Wide range of data, esp. human and monkey 
1 General info for laymen w/ interest in NASA’s work 
1 Sources for similar experiments, looking for studies on given 
species 

1 Wants full documents/publications listed, more detailed equipment 

info and 

contact info for investigators. 

1 Not useful 

1 If I had a book report on NASA 

1 Documented work on scientific studies that could be used to locale 
supporting research for biological studies 

2 No answer 

c. Any other helpful changes? 

1 A brief oral description of use, and a manual 
1 "Go Back" button at top of scrolling page is poorly placed 
1 Wants keyword search, disliked scrolling selection menus 

1 Would have preferred to explore archive first (problem with 
experiment 
setup) 

1 Criticism of experiment setup as unclear 

1 Fix search tool 

3 No answer 


Computer Experienced Non-Researchers 
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From the Overall Evaluation Form: 

1. How easy was it to understand the interface, on a scale of 1 to 7 (7 hardest)? 

1, 2, 2, 6, 6, 3, 5, 3, 3 

2. Which, if any, parts of those screens were unclear? 

2 Wants longer descriptions; keyword searchable hypertext 

2 All worked fairly well 

1 Got stuck in help when clicking on field definition, had to use window 
menu to go back 

1 When clicking on a specific topic, related experiments didn't automatically 
come up 

1 Search was unclear, entering into fields 
1 Screens were clear but deceptive 

1 No answer 

3. While using the archive, where did you get stuck, lost or hung up? 

1 Document retrieval button fails badly if document doesn't exist 

3 Searching experiments 

2 No answer 

1 Search screens somewhat tough to follow 

1 Unable to retrieve data elements (not present) 

1 Listings of hardware & research subjects 

4. Did you have any trouble using the search screen and search function? 

1 Yes, no keyword search of experiment descriptions and other text; 
hard to remember experiment ID's and mission #'s 

2 No problems 

1 Yes, searching for SLS-1 mission (SLS1 worked) 

3 Search screens somewhat tough to follow 

1 Search was wonderful; very easy to follow 

1 Trouble entering keywords 

5. Please explain any other problems or suggestions you had while using the Data 

Archive. 

1 Larger buttons or text as buttons 

1 Suggests using separate screens instead ol scrolling one big one 
3 No answer 

1 Was only able to get Principal Investigator search to work on first name 
1 No glossary, no data sets 

1 Too many steps to get to data 

1 It would be nice to be able to click on an experiment ID and bring up info on 

that experiment. 

6. Were you able to find data from experiments? 

6 Yes 

3 No 
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7. Were you able to find data easily? 

1 No, no keyword searches 

4 Yes 

3 No, no data sets 

8. What about the data presentation was difficult to understand? 

1 Too many code numbers for missions and experiments, too little description 
1 Some prefixes were difficult to understand 

1 Data is unorganized 

1 No glossary, no data sets 

1 Criticism of experiment setup 

4 No answer 

9 . While browsing the catalogs what buttons and/or functions did you use? 

1 All 

1 Liked "Go Back” a lot, very useful 

1 Search, help, sort, home, go back, find all 

1 Mouse, arrow keys (!) 

1 Home, go back - make easier to locate on all pages 
1 Scroll arrows 

1 Mission, subjects, biospecimens, glossary, experiments, documents, 

hardware, personnel, acronyms, help. 

1 Sort, acronyms, glossary 

1 No answer 

10. Of those buttons and/or functions, which were not clear? 

1 Documents and Sessions 

1 No glossary; suggests combination of categories 

2 No glossary 

1 Add "Go Back” button to bottom of long screens 

2 None - all pretty self-explanatory 

1 Learning curve on the interface 

1 No answer 


From the Content Suitability Form: 

A. a. How useful is this type of information to your work on a scale of 1 to 7 (7 most 
useful)? 

4, 6, 2, 2, 5, 3, 1, 4, 6 

b. Is the content level right for your background (1 - 7, 7 most useful)? 

2, 5, 5, 3, 5, 4, 1, 4, 6 


c. Please elaborate: 

Answers not statistically relevant here 



1 More graphs, data sets 

2 Data not useful for subject's work (not researcher) but seems useful 
given general level of biological knowledge 

1 Non-life-science student not familiar with some terminology 
1 Not the kind of information used regularly, but if the need arose 
it would be very useful and easy to use 
1 Experiment descriptions were excellent 

d. Is there other information you would like to see provided... 

(Scale of 1 - 7, 1 is too little, 7 is too much) 

Overall 

2, 4, 5, 5, 4, 4, 3. 6, 3 

Experiment descriptions 

3, 4, 4, 5, 4, 4, 4, 5, 3 

Hardware descriptions 

2. 4. 2, 6, 2, 4, 4, 6, 3 

Document descriptions 

2, 5, 4. 4, 2. 4, 4. 5, 3 

Data set/element descriptions 

2, 6. 4, 4, 2, 4, 4, n/a, 3 

Other desired 

5 No answer 

l Experimental abstracts; ways to contact related people and 
organizations 

1 Less technical material for those with not too much 
biological 
background 

1 Wanted it easier to get additional info on categories near the 
bottom 
of the page 

1 Wants glossary and data sets 

e. Any additional comments 

1 Keyword search; forward screen button 

2 Much better than on-line version 

1 Improve help system 

3 No answer 

1 Excellent content; should be easier to move page-to-page 

1 No. 

B. a. How well organized is the information on the Excel spreadsheets? 

(Scale of 1 - 7, 1 is not organized, 7 is well organized) 

n/a, 4, 6, 5, 5, 4, n/a, n/a, 6 

b. How understandable is the information and data on the spreadsheets? 

(Scale of 1 - 7, 1 is not clear, 7 is very clear) 
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n/a, 4, 6, 3, 6, 4, n/a, n/a, 6 

c. Any additional comments on spreadsheet data? 

4 No answer 

4 No 

l Would be nice if text boxes were smaller to reduce scrolling 

C. a. Are total data useful for scientific work? 

(Scale of 1 - 7, 1 is not useful, 7 is highly useful) 

1, 5, 5, 5, 6, 4, 5. 5, 6 

Comments 

1 Data insufficiently quantified for citation purposes; needs raw 
data; wants to be able to search for experiments w/o needing ID. 

2 Thinks archive would be useful in getting information about prior 
experiments in a given area 

1 Thinks it would be great for people who could understand it 

1 Watching the "movie" was very useful 

2 Would be useful for scientists once interface learned 

1 Anyone looking for details on a subject they already had some 

knowledge of would find it very useful 

b. How might data be useful to tester? 

1 As a sign of what's being done in the field; as a pointer to "real info" 
such as journal papers 

1 As a novice, exploring what NASA has been doing in life sciences 
1 Examining hardware used in experiments 

1 If experiment mix included atmospheric experiments 
4 Useful research tool 

1 Would be useful if subject was a science major 

c. Any other helpful changes? 

1 Sessions button doesn't work; search screens badly laid out 

2 No Answer 

1 Get the on-line version working 

1 Expand subject range 

2 Improve ability to navigate 

1 Glossary and data sets 

1 Add data sets 


Computer Inexperienced Non-Researchers 
From the Overall Evaluation Form: 

1. How easy was it to understand the interface, on a scale of 1 to 7 (7 hardest)? 
3, n/a, 1, 6, 1, 2, 4, 2, 4 
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2. Which, if any, parts of those screens were unclear? 

1 Confusion between mission and payload ID 

2 Search screens 

2 None 

1 Glossary and acronyms 

1 Navigation initially confusing 

2 No answer 

3. While using the archive, where did you get stuck, lost or hung up? 

1 No answer 

1 Search and sort 

3 Search 

1 Terminology was confusing 

1 Trying to retrieve experimental data 

1 Missions not clear; unable to get raw data 

1 Seeking answers to compulsory questions 

4. Did you have any trouble using the search screen and search function? 

1 The acronyms; would like to be able to winnow search lists 
by typing first letter 

2 No answer 

1 Yes; pop up menus are inconsistent, sometimes they are there and 
sometimes 

they are not. 

4 No 

1 Too much data required to do a search 

5. Please explain any other problems or suggestions you had while using the Data 
Archive. 

1 Difficult at first, but improves with experience 

2 No answer 

1 Difficult to find specific info 

1 Problems with most everything 

1 No . 

1 Home didn't return to first page; cancelling took subject out ot system 

altogether. Glossary didn't work. 

1 Deciphering acronyms 

1 Figuring out where to go to look for specific information 

6. Were you able to find data from experiments? 

4 Yes 

1 Yes, but not raw data 

1 No answer 

3 No, not experimental data 

7. Were you able to find data easily? 

3 Yes 
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1 No answer 

5 No 

8. What about the data presentation was difficult to understand? 

3 No answer 

2 Unable to find data sets (not present) 

1 Help wasn't helpful 

1 None. 

1 Use of ID#’s to reference experiments 

1 Where to seek specific info 

9. While browsing the catalogs what buttons and/or functions did you use? 

1 View all items 

1 No answer 

1 Searching difficult since info is spread out through multiple 
catalogs; a .arch covering all catalogs would be useful 

2 All 

1 Go back, scroll, all front page buttons except help. 

1 Tried Glossary, home, help - Glossary and home didn't work well 

1 Mission, Experiment, Search 

1 All except Biospecimens, Personnel 

10. Of those buttons and/or functions, which were not clear? 

1 Hardware and data sets 

3 All were clear 

1 Glossary (didn't work) and acronyms 

4 No answer 

From the Content Suitability Form: 

A. a. How useful is this type of information to your work on a scale of 1 to 7 (7 most 
useful)? 

3. 2, 1, 1, 1, 2, 2, 1, 1 

b. Is the content level right for your background (1 - 7, 7 most useful)? 

3, 2. 1, 4, 3. 1, 2, 3. 1 

c. Please elaborate: 

Answers not statistically relevant here 

1 Subject is cherri. major, so field relevance is marginal at best 

4 Life sciences not subject's field 

3 Subject not scientist 

1 Data sets missing 

d. Is there other information you would like to see provided... 

(Scale of 1 - 7, 1 is too little, 7 is too much) 

Overall 
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4, 5, 4, 4, 5, 2, 6, 4, 2 


Experiment descriptions 

4, 5, 4. 5, 4, 2, 4, 5, 2 

Hardware descriptions 

4, 5, 2, 4, 4. 5, 4, 5, 3 

Document descriptions 

3, 1, 2, 4, 5, 2, 4, 1, 4 

Data set/element descriptions 
3. 1, 2, 4. 4, 3. 4, 4, 5 

Other desired 

3 No answet 

1 Clearer disniay of results and data 

1 Subject was unable to retrieve Personnel role descriptions 

2 Data sets 

1 Good overall, high level of detail 

1 For someone not going into chemistry, lab science, or a 
similar 

discipline, it should he less technical 
e. Any additional comments 
3 No answer 

3 Make it more appealing to non-scientists 

1 Information content is overwhelming 

1 Subject put off by numbers, wanted simple subject-based search 
1 Acronyms were very confusing, should be accessible directly from 
text itself. 

B . a. How well organized is the information on the Excel spreadsheets? 

(Scale of 1 - 7, 1 is not organized, 7 is w-ell organized) 

5, n/a, 4, 1,6, n/a, n/a, n/a, n/a 

b. How understandable is the information and data on the spreadsheets? 

(Scale of 1 - 7, 1 is not clear, 7 is very clear) 

5, n/a. 3, 2, 3, n/a, n/a, n/a, n/a 

c. Any additional comments on spreadsheet data? 

7 No answer 

1 Lump data together to reduce back-and-forth navigation 

1 No 

C. a. Are total data useful for scientific work? 

(Scale of 1 - 7, 1 is not useful, 7 is highly useful) 

5, 6, 5, 4, 6, 3, n/a, 3, 5 
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Comments 


1 Should be easier to locate a specific experiment by name instead of 
by ID number 

1 If information in files was insufficient, there were links to other 
documents 

that would contain the needed information 

2 Subject stresses lack of scientific background 

1 Too complex for average person to use 

2 Wants data sets, better search facilities 

1 Subject unsure due to being neither a bio major nor a researcher 

1 Subject thinks it would be useful to persons in more applicable 

fields 

b. How might data be useful to tester? 

1 In research 

2 General knowledge source 

1 Probably not useful to subject, who isn't bio or researcher 
1 Not at all, given that subject is in marketing 
1 It would be more useful if data contents were geared more towards 
weather patterns as seen from space, since subject is in meierology 

3 No answer 

c. Any other helpful changes? 

4 No answer 

1 Movie was way cool 

1 Data sets 

1 Make all functions work 

1 Be able to open more than one window at a time; click on keywords 
to 

get definitions; improve font used for passage text. 

1 Reduce technical complexity to make it easier on the "average 
person" 
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